“Stealth Crawlers” Are Not a Threat to the Open Web. Bills Targeting Them Would Be.

“Stealth Crawlers” Are Not a Threat to the Open Web. Bills Targeting Them Would Be.

There’s a new boogeyman in the battles over AI: so-called “stealth crawlers.” We’ll admit it—the term “stealth crawlers” sounds quite nefarious. In reality, they’re anything but.

“Stealth crawlers” are simply automated tools to access and collect public web data—without disclosing the user’s identity. Private crawlers like these facilitate all kinds ofimportant workthat benefits the public, including investigative reporting, academic research, cybersecurity protection, and more.

Anonymous crawling enables some of the most publicly beneficial uses of the open web.

Many publishers want to unmask crawlers anyways—and are pushing for new legislation that would give them new powers to do so. These legislative proposals threaten the open web, user privacy, and valuable research without directly addressing the problems they’re supposedly intending to solve.

Alarmingly, these harmful proposals are gaining traction. The New York state legislature has already passed such a bill, theNY Stealth Crawler Protection Act, which is now on Governor Hochul’s desk. We expect to see similar bills introduced in other states, and potentially in Congress. That’s a big problem for the open web—and the many benefits it provides.

Anonymous crawling is worth protecting

Anonymous crawling enables some of the most publicly beneficial uses of the open web. Researchers, journalists, and other watchdog groups use unidentified automated tools to gather the information necessary to hold powerful institutions accountable and protect the public.

Anonymous crawling fuels important investigative journalism. For example,The Markup, a non-profit news site, usedanonymous crawlersto investigate potentially anti-competitive practices by tech companies, such as Amazon’s tendency toprioritize Amazon brands and Amazon-exclusive productsover competitors with higher ratings. The crawlers identified themselves asordinary Firefox browsersto web servers, which allowedThe Markupto understand how Amazon search results pages would appear to ordinary users. Similarly,ProPublicaused an automated tool designed to simulate an ordinary Amazon customer to reveal that thesite steered shoppers to more expensive productsover cheaper alternatives.

Anonymous web scraping is also crucial forcybersecurity professionals, who use automated tools to monitor the web for information that helps them protect against malicious attackers.Privacy tools, including EFF’s ownPrivacy Badger, also crawl sites anonymously to identify trackers without compromising user privacy.

However, without the ability to scrape anonymously, these tools would likely be blocked. Sites can—and do—block crawlers operated by researchers, journalists, and activists who criticize them. For example, Facebookshut downaccounts belonging to researchers who used automated tools to study misinformation on the platform and demanded that theytake downpublished research. Many sites block automated access by anyone who hasn’t paid to crawl public webpages.

Unmasking crawlers threatens the open web

News publishers—and their allies in government—say thatunmasking crawlers is necessary to protect news organizations from technological strain caused by AI-related crawling, and fears that AI could reduce news sites’ traffic and ad revenue. These are legitimate concerns.

But enacting broad, reactionary restrictions on automated access is not the answer.Legislation targeting anonymous crawlingthreatens the open web, user privacy, and valuable research without actually addressing these technological and potential economic harms of scraping.

The New York state legislature recently passed theNY Stealth Crawler Protection Act, a law that would make it illegal to crawl news websites without revealing who is operating the crawler and all possible future uses of the data collected by the crawler. The law would give websites the power to obtain court orders that unmask anyone using an unidentified crawler—without any evidence that they broke the law.

Laws like the New York bill sweep far beyond AI, and do not meaningfully address the technological or potential harms of AI-related web scraping. These policies would chill beneficial crawling by allowing publishers to veto lawful public access, giving them the power to block not just bad actors, but also security professionals, researchers, dissidents, or anyone who has not paid for a license to view public text. This needlessly undermines the free and open internet.

Digital news publishers—like most websites—face real technological challenges in the AI era. While web crawling has been around for decades, with the proliferation of AI, crawlers now collect far more public web data than they used to. This pushes servers closer to their maximum capacity, and if some bots collect information too aggressively, they may strain web servers to the point that it degrades site performance. The problem is not anonymity—so unmasking crawlers won’t solve it. The real problem is overaggressive crawling, which can be effectively addressed with technical measures that target harmful conduct without impeding anonymous access to information.

A better path forward

There are other, far less harmful ways to protect publishers from the harms these “stealth crawler” laws claim to target. Addressing the harms of AI-related crawling requires policies that narrowly target the causes of these issues–without undermining free expression and the open web. Policies that target crawlers and scrapers are anything but.

Related Issues

Related Updates

EFF joined a group of 18 civil society organizations to send a letter encouraging New York Governor Kathy Hochul toSenate Bill 9934A, the New York Stealth Crawler Prohibition Act. The letter states:While framed as a measure to protect local journalism, this legislation harms free expression and establishes a...

Many in Silicon Valley, and in U.S. business at large, seem to believe innovation springs only from competition, a race to build the next big thing first, cheaper, better, best. But what if collaboration and community breeds innovation just as well as adversarial competition?...

This appeal concerns the right of internet users to speakanonymously.Plaintiffs are attempting to require Reddit to reveal information about specific users who discussed downloading copyrighted material, on the theory that they may have evidence relevant to Plaintiffs’ copyright claims against an ISP. EFF’s amicus brief...

Related Issues

Back to top

Follow EFF:

Check out our 4-star rating onCharity Navigator.

Contact

About

Issues

Updates

Press

Donate